Method for producing a digital photo wherein at least some of the pixels comprise position information, and such a digital photo

ABSTRACT

Method for producing a digital photo comprising pixels, wherein at least some of the pixels comprise position information, the method comprising of: taking a photo with a known geometry of the optical system with which the photo is taken; recording the position from which the photo has been taken; recording the direction in which the photo has been taken; providing a three-dimensional model comprising points which are comprised by at least an outer surface of an object in the field of view of the position where the photo has been taken, wherein the points comprise position information; relating a pixel on the one hand and a corresponding point in the three-dimensional model on the other; and recording, in a manner associated with the pixel, the position information of the corresponding pixel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This national stage patent application under 35 U.S.C. § 371 claimspriority to PCT application no. PCT/NL2011/050470 filed Jun. 29, 2011,which claims priority to Netherlands patent application no. 2004996filed Jun. 29, 2010, the disclosures of each of which are incorporatedherein by reference for all purposes.

The present invention relates to a method for producing a digital photocomprising pixels, wherein at least some of the pixels comprise positioninformation.

The present invention also relates to such a digital photo.

Applicant is engaged in the collection of digital images of theinfrastructure and supplies this information to governments, estateagents, utility companies, banks, insurance companies and others for thepurpose of evaluating local situations, for instance the location ofproperty, the state of infrastructural facilities and local trafficsituations. For this purpose applicant travels structured routes withcars having cameras mounted on the roof. As the vehicle drives along thecameras take photos of the surrounding area at fixed distances. The carsare equipped with a position-determining system in order to record thepositions from which the photos are taken. A problem of the digitalphotos obtained in this manner is that in some situations they do notprovide sufficient information about the position of pictured objects.Maps or drawings of the area, for instance cadastral maps, are thereforeused in addition to the digital photos.

The present invention has for its object to provide a photo from whichposition information can be derived.

The present invention provides for this purpose a method for producing adigital photo comprising pixels, wherein at least some of the pixelscomprise position information, the method comprising of: taking a photowith a known geometry of the optical system with which the photo istaken; recording the position from which the photo has been taken;recording the direction in which the photo has been taken; providing athree-dimensional model comprising points which are comprised by atleast an outer surface of an object in the field of view of the positionwhere the photo has been taken, wherein the points comprise positioninformation; relating a pixel on the one hand and a corresponding pointin the three-dimensional model on the other; and recording, in a mannerassociated with the pixel, the position information of the correspondingpixel.

Position information of a pixel of the photo is obtained by calculatingon which pixel a point with position information from thethree-dimensional model is imaged. The position from which the photo istaken is known, and therefore also the corresponding position in thethree-dimensional model. Since the direction in which the photo is takenis recorded, and the geometry of the optical system with which the photois taken and the pixel configuration of the photo is known, it ispossible to determine for each pixel along which line in thethree-dimensional model it is necessary to search for the point whichcorresponds to the pixel imaged in the photo. This line will generallynot run exactly through a point of the three-dimensional model. Acriterion is necessary for this purpose which is less strict than just apoint lying on the line. In a specific embodiment it is possible insteadto search for the point at the shortest distance to the position fromwhich the photo has been taken and wherein the distance to the line liesbelow a threshold value. It is however necessary to be aware that, underthe influence of a possible angular error, the distance to the lineincreases linearly with the distance to the point from which the photohas been taken. In an alternative embodiment points in the vicinity ofthe line are reconstructed to form a plane. The intersection of the lineand the reconstructed plane is then determined.

The three-dimensional model is not limited to three-dimensional modelscomprising only individual points. Any model will suffice from whichpoints can be derived, these points representing an outer surface orouter edge of an object. From a model comprising the outer surfaces ofan object can be derived points which are located in these outersurfaces. These models can therefore be seen as models comprising aninfinite number of points. These models are otherwise preferred tothree-dimensional models comprising a limited number of individualpoints located in the outer surfaces, since the above described “lessstrict” criterion for finding an intersection of the line of sight and apoint in the model is not necessary since a line which passes through asurface always has an exact intersection point. This type of modelprovided with position information is however difficult to obtain inpractice, in contrast to three-dimensional models comprising individualpoints (see below). The present invention therefore also supports modelscomprising individual points.

The direction in which the photo is taken must in principle be recordedin the form of three angles, i.e. an angle in the horizontal plane, forinstance relative to true north, an angle of inclination relative to thehorizontal plane and a rotation angle around the optical axis. Dependingon the use of the camera and/or the construction on which it is mountedin some embodiments, it is safe to assume in specific embodiments thatthe angle of inclination is substantially equal to zero and it sufficesto record only the angle in the horizontal plane. For the sake ofmeasurement precision however, it is recommended to record all threeangles.

In another aspect according to the invention a method is providedfurther comprising the step of generating the three-dimensional model bymeans of a LIDAR device.

LIDAR (Light Detection and Ranging) is a technology similar to radartechnology which is used to measure a contour profile of the Earth'ssurface. A laser beam scans the Earth's surface from an aircraft. Thedistance to the Earth's surface can be derived by measuring the timebetween emitting the laser beam and receiving the reflection. A contourprofile can be constructed by scanning the Earth's surface with thelaser beam and recording the direction in which the measurement is made.If the position of the LIDAR device is also known, it is possible togeocode the contour profile.

The LIDAR technology is suitable for constructing the three-dimensionalmodels for the present invention. The LIDAR information is preferablynot collected from the air but from the ground. Mainly collected fromthe air is information about horizontal surfaces. Relatively fewreflection from the walls of buildings are received from the air,particularly in urban areas with many high-rise buildings. It isrecommended to collect the LIDAR information from the ground from whichthe photos are also taken. Reflections are hereby received especiallyfrom the walls of buildings and other objects, precisely the parts ofbuildings and objects which generally also appear on the photos.

In a further aspect the invention comprises a method further comprisingthe steps of: extracting an object characteristic from the photo;extracting an object characteristic from the three-dimensional model;determining a corresponding pair of an object characteristic from thephoto and an object characteristic from the three-dimensional model;determining a relative position error between the photo and thethree-dimensional model by determining the distance between the positionof an object characteristic in the photo and a corresponding objectcharacteristic from the three-dimensional model; and correcting theposition data of the photo and/or the three-dimensional model on thebasis of the relative position error.

The position information in the three-dimensional model generallycomprises position errors. In addition, a position found will also bemade when determining from where the photo has been taken. The positionerror in the position information of the points in the three-dimensionalmodel will further differ per point. This is because the positions ofthe points are determined from different positions. In for instance anurban area it is recommended to displace the LIDAR device through thedifferent streets so as to thus include information about the differentblocks in the model. The further the point was located from the LIDARdevice, the greater the position error moreover becomes.

Pairs of corresponding object characteristics are found by extractingand comparing object characteristics from both the three-dimensionalmodel and the photo. By subsequently comparing the position of theseobject characteristics in the photo to the corresponding objectcharacteristics from the three-dimensional model it is possible todetermine the relative position error between the photo and thethree-dimensional model. A correction of this position error is thenmade to the position information of the photo and/or thethree-dimensional model.

In a further aspect according to the invention a method is providedwherein correction of the position data comprises of correcting theposition data of both the photo and the three-dimensional model on thebasis of a weighting. It is not known beforehand whether the positionerror is in the position data of the photo or in the position data ofthe three-dimensional model. The position error will in practice be acombination of a position error in both. A correction is made to theposition data of both the photo and the three-dimensional model by meansof a weighting factor.

In a further embodiment the position error is determined by determiningin the photo the angle between a minimum of three characteristics atwhich a corresponding characteristic has been found in thethree-dimensional model. This angle can be determined because thegeometry of the optical system with which the photo has been taken isknown. On the basis of the known positions of the characteristics in thethree-dimensional model it is now possible using resection to determinea position from which the photo has been taken. The position error isdetermined by comparing the thus determined position to the positionrecorded when the photo was taken.

In yet another aspect according to the present invention a method isprovided wherein the weighting is determined by reliability informationof the position determinations of the photo and/or the three-dimensionalmodel. The greater the reliability of the position data of either thephoto or the three-dimensional model, the higher the weighting factorapplied for this position in calculating the optimum position data. In aspecific embodiment at least one of the dilution of precision (DOP)values of the GPS is used in order to determine a reliability. Inanother specific embodiment the position determining system comprises aGPS with an inertial navigation system coupled thereto. The inertialnavigation system provides a position when the GPS cannot determine aposition, for instance as a result of the reception of the satellitesignals between high-rise buildings, and for the purpose of thenecessary accuracy in the position determination. The GPS supplies theinertial navigation system with reference positions for compensation ofposition drift. In this embodiment the reliability information isdetermined on the basis of the period of time since the inertialnavigation system last received a reference position from the GPS. Inyet another specific embodiment use is made of the DOP values as well asthe period of time since receiving a reference position in order todetermine the reliability. In an alternative embodiment the positiondetermination is processed by means of a Kalman filter. In addition toan estimated actual value, the Kalman filter provides an estimate of thequality of the estimated actual value. This estimated quality is in turnused to determine the weighting factor for the position correction.

In a specific aspect according to the invention the intensity of thereflection is also recorded in the LIDAR data. In a further specificembodiment this intensity is used for the purpose, on the basis of thethree-dimensional model in combination with the intensity data, ofgenerating a two-dimensional projection from the viewpoint from whichthe photo has been taken. Corresponding characteristics are then onceagain determined in the photo and the two-dimensional projection, whichdisplay a great measure of visual similarity. Using the correspondingcharacteristics the position error is once again determined, after whichit is corrected, with or without weighting.

In a further aspect according to the present invention a method isprovided wherein the photo is taken from a vehicle while the vehicle isin motion. Traffic is impeded as little as possible during collection ofdata by taking the photo while the vehicle is in motion. In a specificembodiment a plurality of photos are taken in a plurality of directionsusing a plurality of cameras, these photos being stitched to formpanoramic photos. The European patent application EP 1 903 534 ofapplicant describes a technology in which the movement of the vehicle isused to minimize the parallax error.

In an aspect according to the invention a method is provided wherein thevehicle is a car.

According to another aspect according to the invention, a method isprovided wherein the LIDAR device is arranged on a vehicle and whereinthe LIDAR data are collected while the vehicle is in motion.

In yet another aspect according to the invention a method is providedwherein the photo is taken from a vehicle other than the vehicle onwhich the LIDAR device is arranged.

The vehicle from which the photos are taken has the drawback that,depending on the amount of ambient light, the speed may not become toohigh since this results in motion blur in the photo. However, in anurban environment with many high-rise buildings, a high speed has apositive effect on the accuracy of the position determination. Owing tothe high-rise the GPS will regularly be unable to do positiondetermination because the signal from a number of satellites is blockedby buildings. At the moments that a signal is being received fromsufficient satellites, the position will still regularly be inaccuratedue to for instance multipath effects. This is where the inertialnavigation system makes an important contribution by supplying accurateposition information until a moment once again arrives when moreaccurate GPS positions are available. However, because an inertialnavigation system is liable to drift, this may not last too long. A highspeed enhances the position accuracy. A vehicle with a photo camera ishowever limited in its speed. This limitation does not however apply toa vehicle with a LIDAR device. By now having the vehicle with the LIDARdevice travel at the highest possible speed and making use of the abovedescribed method for correcting position errors, a high positionaccuracy is obtained without motion blur occurring in the photos. It ishowever necessary here to take into account the fact that a higher speedproduces a lower point density, since a LIDAR device generally has afixed scanning frequency. In addition, the vehicle with the photo camerais limited to moments of the day when there is sufficient ambient lightto take photos. The vehicle with the LIDAR device can however collectdata 24 hours a day and even has an advantage when it travels at night,since it is likely to be able to travel at higher speed as there is lesstraffic.

The present invention provides a digital photo comprising pixels,wherein each pixel comprises at least one light intensity and at leastsome of the pixels comprise position information. This digital photo hasthe advantage that three-dimensionally measurements are made and spatialdistances can be derived directly from the photo.

In a further embodiment the invention provides a digital photo whereinthe position information comprises a distance from the position fromwhich the digital photo has been taken. Although in an alternativeembodiment the position information comprises three locationcoordinates, the above stated embodiment is more efficient sinceproviding many pixels of three location coordinates results in muchredundant information, and therefore considerably larger files. Bystoring only the distances from the camera position is also possible ata known field of view of the photo to calculate distances in the realworld and between displayed pixels or angles between displayed lines. Ina further embodiment the photo therefore also comprises the field ofview of the photo.

In an alternative embodiment the invention provides a digital photo,wherein the position information comprises a relative position inrelation to the position from which the digital photo has been taken.

In another alternative embodiment the present invention provides adigital photo, wherein the position information comprises a positionrelative to an external coordinate system.

In a further embodiment the present invention provides a digital photo,wherein the photo also comprises a reliability indicator for theposition information. As described above, the reliability indicator ishighly useful in the later correction of position errors.

In a further embodiment the invention provides a digital photo, whereinthe pixels comprise the light intensity for three colour channels. In aspecific embodiment the colour channels comprise a red, a green and ablue colour channel.

In a specific embodiment the invention provides a digital photo, whereinthe photo comprises a field of vision of 360°. This photo can beobtained by making use of a fisheye lens, but can be obtained inalternative manner by stitching together the photos of two or more photocameras.

Further advantages and embodiments are described below with reference tothe accompanying figures, in which:

FIG. 1A is a top view of two vehicles as used in a method according tothe present invention;

FIG. 1B is a perspective view of one of the vehicles of FIG. 1A;

FIG. 2 is a schematic representation of a digital photo for processingto a digital photo according to the invention;

FIG. 3 is a schematic representation of a three-dimensional model asused in a method according to the present invention;

FIG. 4 is a schematic representation of a detail of a digital photoaccording to the invention;

FIG. 5 shows a flow diagram of a method according to the presentinvention; and

FIG. 6 shows a flow diagram of a further method according to the presentinvention.

A vehicle 110 (FIG. 1A) is provided with a first photo camera 112 and asecond photo camera 114. Both cameras are mounted on the roof of the carand have a horizontal field of view of more than 180°, so that theimages of the two photo cameras together produce a complete view aroundvehicle 110. First photo camera 112 is directed forward in the directionof travel. Second photo camera 114 is directed to the rear.

Vehicle 110 is further provided with a position determining device (notshown). In the exemplary embodiment this is a combination of a GlobalPositioning System, GPS, and an inertial position determining system.The inertial position determining system is responsible for accuracy andprovides position information when the GPS is not capable of positiondetermination. The GPS in turn provides for drift compensation in theinertial position determining system by providing reference positions.

Vehicle 110 travels over a road 150 in an area where objects 162, 164,166, 168 are located, such as buildings, traffic signs, bus stops andthe like. During travel the photo cameras 112, 114 take photos of thesurrounding area. For each photo the position where the photo was takenis stored. Photo cameras 112 and 114 are preferably controlled suchthat, when second photo camera 114 takes a photo, the entrance pupil ofsecond photo camera 144 is situated as close as possible to the positionwhere the entrance pupil of first photo camera 112 was situated when ittook a photo at a preceding point in time. It is possible in this way tomerge the photos of photo cameras 112, 114 by means of stitching to forma panoramic photo with the smallest possible parallax error.

The resulting photo 200 (FIG. 2 shows a cropped perspective section)shows objects 162, 164, 166, 168 as recorded by photo camera 112.

A second vehicle 120 is provided with two LIDAR devices 122, 124. LIDARdevices 122, 124 generate a pulsing laser beam. A rotating mirrorreflects the laser beam in a plane 132, 134 perpendicularly of the beamdirection of the laser generator. LIDAR devices 122, 124 are placed suchthat the planes 132, 134 are vertical planes perpendicularly of theground surface on which the vehicle stands, and such that the normal ofthe planes forms an angle of about respectively −45° and 45° to the maindirection of travel of the vehicle in a plane parallel to the groundsurface on which the vehicle stands. The angle ensures that LIDARdevices 122, 124 can also scan behind smaller objects situated alongroad 150. With a scanning direction perpendicularly of the direction oftravel this type of object would form “shadows” behind which largerobjects such as buildings remain partially concealed.

The rotating laser beams reflect on objects and the reflected laserlight is reflected by the rotating mirrors to sensors in LIDAR devices122, 124. The distance to the object is determined on the basis of thepropagation time of the laser pulses from LIDAR device 122, 124 to theobject and back again to LIDAR device 122, 124. Vehicle 120 is furtherprovided with a position determining system in the form of a combinationof a GPS and an inertial position determining system. Because thedirection in which the laser pulse is emitted is further known, theposition of the reflection point can be calculated.

Because vehicle 120 is moving, LIDAR devices 122, 124 scan thesurrounding area and a three-dimensional model 300 (FIG. 3) of the areais formed. The representation in FIG. 3 is from the same point of viewas the photo of FIG. 2. This is however a three-dimensional model andthere is as such no specific point of view linked to the model. Objects162, 164, 166, 168 appear in the three-dimensional model 300 as thepoint clouds 362, 364, 366, 368. Ground surface 370 further appears inthe three-dimensional model.

A digital photo 200 is then provided with depth information (FIG. 4).Each pixel 410 of photo 200, in addition to being provided with thethree conventional colour channels red 412, green 414 and blue 416, isalso provided with a depth channel 418. This depth channel indicates thedistance of the position of the point imaged in pixel 410 to theposition from which the photo was taken. In addition to depth channel418, pixel 410 is further provided with a quality channel 420, thisquality channel 420 indicating the quality or reliability of the depthinformation in depth channel 418.

This process is further described in flow diagram 500 (FIG. 5). Afterstartup 502 of the process a photo 200 is obtained 504 with associatedposition data 506 as described above. Using the LIDAR devices 122, 124LIDAR data are also obtained 508 in the form of propagation times of thelaser pulses in combination with a position of the LIDAR device and adirection in which the laser pulse is emitted. The three-dimensionalmodel 300 is formed 510 on the basis of the LIDAR data, this modelcomprising a point cloud (generally with a plurality of clusters ofpoints, wherein each cluster represents an object 162, 164, 166, 168such as a building).

In a following step a determination 512 is made for each pixel as towhich point in the three-dimensional model 300 corresponds most to thepixel. The distance to the imaged point is then determined 514 on thebasis of the position of the found pixel and the position from whichphoto 200 was taken. Finally, this distance is stored 516 in the pixelin the photo, after which the method is ended.

FIG. 6 shows a method 600 for correcting possible position errors in theposition of photo 200 or the position of the points in three-dimensionalmodel 300. This method can per se be utilized independently to corrector reduce position errors, but is particularly suitable for performingin integrated manner with method 500 in order to increase the accuracyof the depth data. To this end the final step, step 620, of this flowdiagram 600 refers to step 512 in flow diagram 500.

Following startup 602 of the method, as in the above method 500, a photo200 is obtained 604 with associated position information. Visualcharacteristics, such as vertical contrast transitions or angles incontrasting planes, are then extracted 606 from this photo. There is agood chance that such characteristics represent for instance corners ofbuildings or lampposts.

In addition, LIDAR data are also obtained 608 (see also method 500above), on the basis of which a three-dimensional model 300 is generated610. Characteristics such as corners of objects 162, 164, 166, 168 arethen also extracted 612 from the three-dimensional model.

In a subsequent step a search 614 is made for correspondingcharacteristics in photo 200 and three-dimensional model 300. Thecorresponding characteristics are determined by making use of resection.Of the characteristics from photo 200 a provisional position is in thefirst instance determined on the basis of the above described method500. The relative position error is then determined 616 between theposition of the characteristic in three-dimensional model 300 and theprovisional position of the corresponding characteristic in photo 200.

The position of photo 200 and the position of the points of thecharacteristic in the three-dimensional model are then corrected 618 onthe basis of a weighting factor. The weighting factor is determined by areliability indicator of the position determinations of respectivelyphoto 200 and three-dimensional model 300 (or more precisely the pointsin the model corresponding to the characteristic). This reliabilityindicator is for instance determined on the basis of the recordeddilution of precision information of the GPS and how recently theinertial position determining system has been corrected for drift.

Account also has to be taken of the fact that the points inthree-dimensional model 300 are based on measurements taken at differentmoments, and the position errors and reliability indicators are nottherefore the same for all points in model 300.

Once the positions of photo 200 and the points in three-dimensionalmodel 300 have been corrected, this method 600 for correcting positionerrors is ended. As described above, it will then normally be wished todetermine the definitive position of the pixels in photo 200 by applyingsteps 510 to 518.

The embodiments described and shown here are only exemplary embodimentsand serve only to illustrate the present invention. It will beimmediately apparent to the skilled person that many modifications arepossible within the invention. It is thus of course possible to combinethe features of the described and shown embodiments in order to obtainnew embodiments. These new embodiments also fall within the presentinvention. The scope of protection sought is therefore not limited tothe shown and described embodiments but is defined solely by thefollowing claims.

The invention claimed is:
 1. A method for producing a digital photocomprising pixels, wherein at least some of the pixels comprise positioninformation, the method comprising of: a) taking a first photo with aknown geometry of the optical system with which the first photo istaken; b) recording the position from which the first photo has beentaken; c) recording the direction in which the first photo has beentaken; d) using a LIDAR device for providing a three-dimensional modelcomprising points, each point comprising position information, andwherein the points each correspond to a respective reflection point onan outer surface of an object from which reflection point a laser beamfrom the LIDAR device has been reflected back to the LIDAR device; e)for each pixel in the photo: determining, in dependence of the positionand direction recorded for that first photo, a line that runs throughsaid each pixel and the position from which the first photo has beentaken; searching for a point in the three-dimensional model that relatesto the same respective reflection point as said each pixel, based on adistance between such point and the line; if a point in thethree-dimensional model is found that relates to the same reflectionpoint, calculating a distance to be associated with said each pixelusing the distance between the reflection point that corresponds to thepoint found in said three-dimensional model and the position from whichthe first photo has been taken; and storing the distance in associationwith said pixel of said first photo.
 2. The method as claimed in claim1, wherein said searching and finding said point using said linecomprises: determining which point of the three-dimensional model is atthe shortest distance to the position from which the photo has beentaken while the distance from this point to the line lies below athreshold value; or reconstructing a plane using points of thethree-dimensional model that are in the vicinity of the line anddetermining the point as the intersection of the line and thereconstructed plane.
 3. The method as claimed in claim 1, wherein themethod comprises using a first and second photo camera for recordingrespective photos according to step a), and comprises performing step b)and c) for each recorded photo, wherein the digital photo is constructedby stitching the photos obtained by the first and second photo camera.4. The method as claimed in claim 3, wherein the field of view of thedigital photo comprises a field of vision of 360 degrees.
 5. The methodas claimed in claim 4, wherein the first and second camera are mountedon a vehicle, wherein the photos are taken from the vehicle while thevehicle is in motion; wherein the first and second camera are controlledsuch that, when the second photo camera takes a photo, the entrancepupil of the second photo camera is situated as close as possible to theposition where the entrance pupil of the first photo camera was situatedwhen it took a photo at a preceding point in time.
 6. The method asclaimed in claim 1, the method further comprising: extracting acorresponding pair of object characteristics from the photo and thethree-dimensional model; comparing the position of the extracted objectcharacteristics in the photo to the position of the correspondingextracted object characteristics in the three-dimensional model;determining a relative position error between the photo and thethree-dimensional model based on said comparison; and correcting theposition from which the photo has been taken and/or the positioninformation of the three-dimensional model based on said relativeposition error.
 7. The method as claimed in claim 6, wherein correctionof the position comprises correcting the position data of both the photoand the three-dimensional model on the basis of a weighting.
 8. Themethod as claimed in claim 7, wherein the weighting is determined byreliability information of the position determinations of the photoand/or the three-dimensional model.
 9. The method as claimed in claim 5,wherein the vehicle is a car.
 10. The method as claimed in claim 1,wherein the LIDAR device is arranged on a vehicle and wherein the pointsare collected while the vehicle is in motion.
 11. The method as claimedin claim 10, wherein the photo is taken from a vehicle other than thevehicle on which the LIDAR device is arranged.
 12. The method as claimedin claim 1, each pixel comprising a plurality of color channels and adepth channel, in which depth channel said distance is indicated.